Real-world object holographic transport and communication room system

ABSTRACT

A novel holographic transport and communication room system utilizes a single red-green-blue (RGB)-depth (RGB-D) camera to capture the motion of a dynamic target, which is required to rotate around the RGB-D camera, instead of capturing three-dimensional volume of the dynamic target conventionally with a plurality of multi-angle cameras positioned around the dynamic target. The captured 3D volume of the dynamic target subject undergoes relighting, subject depth calculations, geometrical extrapolations, and volumetric reconstructions in a machine-learning graphical transformation feedback loop to synthesize a refined real-time hologram. The resulting hologram in one holographic room system is shared with other users occupying other holographic room systems equipped with similar holographic capabilities for live bilateral or multilateral holographic visualization and collaboration. Preferably, each holographic room system also integrates a mixed-reality content synthesis table for real-time remote participant collaboration in manipulating holographic contents and a one-to-one ratio life-size holographic display and capture tubular device.

INCORPORATION BY REFERENCE

A US non-provisional patent application, U.S. Ser. No. 16/177,328,titled “Mixed-Reality Space Map Creation and Mapping FormatCompatibility-Enhancing Method for a Three-Dimensional Mixed-RealitySpace and Experience Construction Sharing System,” and filed on Oct. 31,2018, is incorporated herein by reference. The present invention is alsoa continuation-in-part application of U.S. Ser. No. 16/177,328 and thusclaims benefit to U.S. Ser. No. 16/177,328.

Furthermore, another US non-provisional patent application, U.S. Ser.No. 16/177,319, titled “Electronic System and Method forThree-Dimensional Mixed-Reality Space and Experience Construction andSharing,” and filed on Oct. 31, 2018, is also incorporated herein byreference. The present invention is also a continuation-in-partapplication of U.S. Ser. No. 16/177,319 and thus claims benefit to U.S.Ser. No. 16/177,319.

In addition, another US non-provisional patent application, U.S. Ser.No. 16/699,280, titled “Surrogate Visitor Mixed-Reality Live EnvironmentSharing System with Remote Visitors,” and filed on Nov. 29, 2019, isalso incorporated herein by reference. The present invention is also acontinuation-in-part application of U.S. Ser. No. 16/699,280 and thusclaims benefit to U.S. Ser. No. 16/699,280.

BACKGROUND OF THE INVENTION

The present invention generally relates to mixed-reality environmentvisualizations and interactive immersive bilateral or multilateralholographic communications. In particular, the present invention relatesto real-time holographic object and live mixed-reality environmentsharing between remotely-located and dedicated holographic communicationrooms connected by one or more high-bandwidth wireless communicationprotocols. The present invention also relates to real-time and livemixed-reality environment sharing between a physical visitor and remotevirtual visitors to a physical landmark, wherein the remote virtualvisitors are provided with computer graphics-generated real-timemixed-reality environments through the physical visitor's perception toparticipate in the immersive and interactive visitor experiences to thelandmark.

More specifically, the present invention relates to holographictransport and communication room-based holographic mixed-reality (HMR)live environment sharing system among a plurality of such dedicatedholographic transport and communication rooms that incorporatespecialized holographic display and capture tubes and/or specialized HMRdisplay tables. The present invention also relates to a method ofproviding the surrogate visitor-driven HMR live environment sharing toremotely-located virtual visitors. In addition, the present inventionalso relates to immersive mixed-reality visualization of real physicaland holographic elements in a designated real physical space.

Virtual reality (VR) and augmented reality (AR) applications are gainingincreasing popularity and relevance in electronic user applications. Forexample, VR headsets for computers and portable devices are able toprovide interactive and stereoscopic gaming experiences, trainingsimulations, and educational environments for users wearing the VRheadsets. In another example, augmented reality (AR) mobile applicationsare designed to add texts, descriptions, or added (i.e. “augmented”)digitized materials to physical objects if a user wears AR goggles orutilizes AR-compatible mobile applications executed in portable devices.For one of ordinary skill in the art, virtual reality (VR) refers to acompletely computer-generated synthetic environment with no directcorrelations to a real physical space or a real physical object, whileaugmented reality (AR) refers to descriptive digital materials that aredisplayed next to a machine-recognized real physical object to add or“augment” more information to the physical reality.

However, conventional VR and AR applications are unable to provideseamless integration of ultra-high resolution and lifelike holographicthree-dimensional objects juxtaposed to real physical objects located ina particular physical location for interactive and immersive curationwith both synthetic and real objects, because the conventional VRapplications merely provide user interactions in a purelycomputer-generated synthetic (i.e. virtual) environment with nocorrelation to physical objects in a real physical space, while theconventional AR applications merely provide additional informationaloverlays (i.e. information augmentation) to machine-recognized realphysical objects via partially-transparent AR goggles or AR-enabledcamera applications in mobile devices.

A recent evolution of conventional VR and AR applications has resultedin an innovative intermixture of computer-generated lifelike holographicobjects and real objects that are synchronized and correlated to aparticular physical space (i.e. as a “mixed-reality” (MR) environment)for immersive user interactions during the user's visit to theparticular physical space. Unfortunately, actual implementations of themixed-reality (MR) environment for particular physical spaces related totourist landmarks or multi-party holographic visual communications haveencountered some practical limitations and shortcomings.

For example, typical head-mounted displays (HMDs), conventionalholographic image-capture cameras positioned around a target forhologram generation, and corresponding graphics processing computerservers are very expensive, and often require a multi-million dollarbudget to create even one studio capable of MR environment-basedholographic communications. Therefore, high expenditure requirements forproviding mixed-reality (MR) experience have been an impediment to massadoption of multi-way and real-time holographic communications.Furthermore, the logistics of providing the MR environments to a largenumber of users for multilateral holographic communications is often toodifficult and bottlenecked due to a limited number of available HMDequipment, space confinements, and safety or injury risks.

Therefore, it may be advantageous to provide a novel electronic systemand a related method of operation that reduce the logistical complexityand bottlenecks for providing mixed-reality environments to a largenumber of participants in real-time by reducing the number of necessaryholographic camera and graphic processing server equipment, withoutsacrificing the quality of hologram synthesis and sharing with aplurality of holographic communication participants.

Furthermore, it may also be advantageous to provide a scalable, modular,cost-effective, and standardized holographic transport and communicationspace, which can be replicated in various remote locations formulti-party hologram communications in a shared mixed-reality (MR)environment.

Moreover, it may also be advantageous to provide an electronic systemthat supports the scalable, modular, cost-effective, and standardizedholographic transport and communication space, which also accommodatesone or more user interaction designers for the multi-party hologramcommunications in the shared MR environment

In addition, it may also be advantageous to provide a novel method forenhancing mixed-reality space map creation and mapping formatcompatibilities among various three-dimensional mixed-reality space andexperience construction platforms to promote pervasive sharing ofnumerous mixed-reality environments and contents created by a pluralityof mixed-reality experience designers across seemingly-incompatible mapvisualization standards.

SUMMARY

Summary and Abstract summarize some aspects of the present invention.Simplifications or omissions may have been made to avoid obscuring thepurpose of the Summary or the Abstract. These simplifications oromissions are not intended to limit the scope of the present invention.

In one embodiment of the invention, a real-world object holographictransport and communication room system is disclosed. This systemcomprises: (1) a holographic transport and communication room with avertical wall; (2) a hologram bilateral monitoring device mounted on thevertical wall; (3) a single red-green-blue (RGB)-depth (RGB-D) camerainstalled near the hologram bilateral monitoring device, wherein thesingle RGB-D camera captures real-time z-axis depth parameters of atarget object, in addition to conventional RGB color data; (4) thetarget object standing and self-rotating 360-degrees at least once infront of the single RGB-D camera to enable the single RGB-D camera tocapture three-dimensional (3D) volume information of the target objectover a specified duration; (5) a graphics server receiving a continuousstream of the 3D volume information of the target object over thespecified duration while the target object is self-rotating 360-degreesat least once in front of the single RGB-D camera, wherein the specifiedduration of the continuous stream of the 3D volume information providessufficient time-variable volumetric information of the target object tocreate, sharpen, and display a computerized hologram of the targetobject by the graphics server in a real-time bilateral holographiccommunication with a remote user outside the holographic transport andcommunication room; (6) a mixed-reality (MR) headset worn by a localuser located inside the holographic transport and communication room;(7) a remote hologram from the remote user projected in the holographictransport and communication room, wherein the remote hologram from theremote user is visible through the MR headset worn by the local userinside the holographic transport and communication room; and (8) anautostereoscopic holographic display and capture tubular device thatdoes not require a separate headset gear to visualize the remotehologram for other local users in the holographic transport andcommunication room.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows a process flow for a surrogate visitor holographicmixed-reality (HMR) live environment sharing system withremotely-located visitors, in accordance with an embodiment of theinvention.

FIG. 2 shows a system block diagram for a surrogate visitor HMR liveenvironment sharing system with remotely-located visitors, in accordancewith an embodiment of the invention.

FIG. 3 shows an exemplary application diagram for a surrogate visitorHMR live environment sharing system with remotely-located visitors, inaccordance with an embodiment of the invention.

FIG. 4 shows an example of mixed-reality space mapping formatcompatibility-enhancing dual-file structures for a 3D mixed-realityspace and experience construction sharing system that can also beutilized for a surrogate visitor HMR live environment sharing system, inaccordance with an embodiment of the invention.

FIG. 5 shows a mixed-reality space map creation and mapping formatcompatibility-enhancing method flowchart for a 3D mixed-reality spaceand experience construction sharing system, in accordance with anembodiment of the invention.

FIG. 6 shows a process flow diagram for a three-dimensional (3D)mixed-reality space and experience construction sharing system (i.e. the“HoloWalks” system) that can also be utilized for a surrogate visitorHMR live environment sharing system, in accordance with an embodiment ofthe invention.

FIG. 7 shows a system block diagram for a three-dimensional (3D)mixed-reality space and experience construction sharing system (i.e. the“HoloWalks” system) that can also be utilized for a surrogate visitorHMR live environment sharing system, in accordance with an embodiment ofthe invention.

FIG. 8A shows a HoloWalks experience designer walking through a physicalspace, which enables the HoloWalks system to automatically andintelligently create three-dimensional (3D) digitized mappingvisualization data for synthesizing a mixed-reality artificial layerconstruction, in accordance with an embodiment of the invention.

FIG. 8B shows a HoloWalks experience designer selecting a desired spotwithin the three-dimensional (3D) map in a mixed-reality visualinterface to initiate a mixed-reality holographic guide contentcreation, in accordance with an embodiment of the invention.

FIG. 8C shows a HoloWalks experience designer placing holographiccontents and overlaying user interaction elements within the 3D map inthe mixed-reality visual interface in the HoloWalks system, inaccordance with an embodiment of the invention.

FIG. 8D shows a user interacting with a holographic guide and/orcontents in a mixed-reality environment provided by a HoloWalks viewer,in accordance with an embodiment of the invention.

FIG. 9 shows an example of multiple mixed-reality (MR) artificial layerssuperimposed on a physical space for construction of a mixed-reality(MR) application from the three-dimensional (3D) mixed-reality space andexperience construction sharing system (i.e. the “HoloWalks” system), inaccordance with an embodiment of the invention.

FIG. 10A shows a first step in a mixed-reality (MR) application example,in which a user perceives physical objects in a physical space, inaccordance with an embodiment of the invention.

FIG. 10B shows a second step in the mixed-reality (MR) applicationexample, in which a mixed-reality (MR) experience designer wears ahead-mounted display (HMD) in the same physical space to initiate an MRcontent creation, in accordance with an embodiment of the invention.

FIG. 10C shows a third step in the mixed-reality (MR) applicationexample, in which the HoloWalks system enables automated intelligentthree-dimensional (3D) mapping of the physical space via HMD spacescanning by the MR experience designer, in accordance with an embodimentof the invention.

FIG. 10D shows a fourth step in the mixed-reality (MR) applicationexample, in which the HoloWalks system completes user glare-invokedintelligent 3D mapping of the physical space, in accordance with anembodiment of the invention.

FIG. 10E shows a fifth step in the mixed-reality (MR) applicationexample, in which the HoloWalks system creates virtual coordinates onmixed-reality artificial layer(s) in preparation of the MR experiencedesigner's MR content synthesis, in accordance with an embodiment of theinvention.

FIG. 10F shows a sixth step in the mixed-reality (MR) applicationexample, in which the MR experience designer selects and directsmixed-reality objects (MROs) and interactions in the MR artificiallayer(s) intertwined with physical objects and physical space, inaccordance with an embodiment of the invention.

FIG. 10G shows a seventh step in the mixed-reality (MR) applicationexample, in which the MR experience designer places and directs moreMROs and interactions in the MR artificial layer(s) intertwined withphysical objects and physical space, in accordance with an embodiment ofthe invention.

FIG. 10H shows an eighth step in the mixed-reality (MR) applicationexample, in which the MR experience designer places more MROs,mixed-reality holograms (MRHs), and interactions in the MR artificiallayer(s) intertwined with physical objects and physical space, inaccordance with an embodiment of the invention.

FIG. 10I shows a ninth step in the mixed-reality (MR) applicationexample, in which an MR experience viewer equipped with HMD engages inlifelike intertwined visualization of MROs, MRHs, and physical objectsin the same physical space, in accordance with an embodiment of theinvention.

FIG. 10J shows a tenth step in the mixed-reality (MR) applicationexample, in which the MR experience viewer, while not wearing the HMD,only sees physical objects without visual recognition of the MROs andthe MRHs implemented in the same physical space in the MR artificiallayer(s), in accordance with an embodiment of the invention.

FIG. 11 shows a single camera and machine learning-based holographicimage capture example, in accordance with an embodiment of theinvention.

FIG. 12 shows a target object-initiated self-rotation around the singlecamera and machine learning apparatus for hologram generation, inaccordance with an embodiment of the invention.

FIG. 13 shows cost and convenience advantage of the single camera andmachine learning-based holographic image capture method, in accordancewith an embodiment of the invention.

FIG. 14 shows a novel real-world object holographic transport andcommunication room system configuration, in accordance with anembodiment of the invention.

FIG. 15 shows another novel real-world object holographic transport andcommunication room system configuration, in accordance with anembodiment of the invention.

FIG. 16 shows a two-room application example of the novel real-worldobject holographic transport and communication room system, inaccordance with an embodiment of the invention.

FIG. 17 shows a system component diagram for the two-room applicationexample of the novel real-world object holographic transport andcommunication room system, in accordance with an embodiment of theinvention.

DETAILED DESCRIPTION

Specific embodiments of the invention will now be described in detailwith reference to the accompanying figures. Like elements in the variousfigures are denoted by like reference numerals for consistency.

In the following detailed description of embodiments of the invention,numerous specific details are set forth in order to provide a morethorough understanding of the invention. However, it will be apparent toone of ordinary skill in the art that the invention may be practicedwithout these specific details. In other instances, well-known featureshave not been described in detail to avoid unnecessarily complicatingthe description.

The detailed description is presented largely in terms of description ofshapes, configurations, and/or other symbolic representations thatdirectly or indirectly resemble one or more electronic systems andmethods for a novel real-world object holographic transport andcommunication room system. These process descriptions andrepresentations are the means used by those experienced or skilled inthe art to most effectively convey the substance of their work to othersskilled in the art.

Reference herein to “one embodiment” or “an embodiment” means that aparticular feature, structure, or characteristic described in connectionwith the embodiment can be included in at least one embodiment of theinvention. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment. Furthermore, separate or alternative embodiments arenot necessarily mutually exclusive of other embodiments. Moreover, theorder of blocks in process flowcharts or diagrams representing one ormore embodiments of the invention do not inherently indicate anyparticular order and do not imply any limitations in the invention.

One objective of an embodiment of the present invention is to provide anovel electronic system and a related method of operation that reducethe logistical complexity and bottlenecks for providing mixed-realityenvironments to a large number of participants in real-time by reducingthe number of necessary holographic camera and graphic processing serverequipment, without sacrificing the quality of hologram synthesis andsharing with a plurality of holographic communication participants.

Furthermore, another objective of an embodiment of the invention is toprovide a scalable, modular, and cost-effective holographic transportand communication space as a standardized room with supportingholographic equipment, which is configured to be replicated in variousremote locations for multi-party and standardized room-to-room hologramcommunications in a shared mixed-reality (MR) environment.

In addition, another objective of an embodiment of the invention is toprovide an electronic system that supports the scalable, modular,cost-effective, and standardized holographic transport and communicationspace, which also accommodates one or more user interaction designersfor the multi-party hologram communications in the shared MRenvironment.

Another objective of an embodiment of the present invention is toprovide a novel method for enhancing mixed-reality space map creationand mapping format compatibilities among various three-dimensionalmixed-reality space and experience construction platforms to accommodateconvenient and seamless sharing of numerous mixed-reality environmentsand contents created by a plurality of mixed-reality experiencedesigners across seemingly-incompatible map visualization standards.

In addition, another objective of an embodiment of the present inventionis to provide a novel electronic system that enables an intermixture ofcomputer-generated lifelike holographic objects and real objects thatare synchronized and correlated to a particular physical space (i.e. asa “mixed-reality” (MR) environment) for immersive andvividly-interactive user experiences during the user's virtualized“remote” visit to the particular physical space through a real-timeholographic mixed-reality live environment sharing.

Furthermore, another objective of an embodiment of the present inventionis to provide a novel electronic system that accommodates a userinteraction designer to construct and configure a mixed-reality (MR)environment and various potential user interactivities for a geographiclandmark, a museum, or another tourist destination, and subsequentlyshares the MR environment with other user interaction designers andremotely-located users (e.g. remotely-located virtual tourists) via thesurrogate visitor-driven holographic mixed-reality (HMR) liveenvironment sharing. In context of various embodiments of the invention,this user interaction designer is referred to as a “mixed-reality (MR)experience designer,” or a “surrogate visitor” for a three-dimensionalmixed-reality space and experience construction sharing system (i.e. the“HoloWalks” system) and a surrogate visitor HMR live environment sharingsystem.

Moreover, another objective of an embodiment of the present invention isto provide a method of operating the three-dimensional mixed-realityspace and experience construction sharing system (i.e. the “HoloWalks”system) for MR experience designers and MR experience viewers (e.g.tourists, visitors, etc.) focused on a particular geographic landmark, amuseum, or another tourist destination.

For the purpose of describing the invention, a term referred to as“mixed reality,” or “MR,” as an acronym, is defined as an intermixtureof computer-generated lifelike holographic objects and real physicalobjects that are synchronized and correlated to a particular physicalspace for immersive user interactions during the user's visit to theparticular physical space. Typically, unlike conventional augmentedreality applications, the computer-generated lifelike holographicobjects are ultra high-resolution (e.g. 4K/UHD) or high-resolution (e.g.HD quality or above) three-dimensional synthetic objects that areintermixed and/or juxtaposed to real physical objects, wherein a viewerimmersed in the mixed-reality environment is often unable to distinguishthe synthetic nature of the computer-generated lifelike holographicobjects and the real physical objects provided by the mixed-realityenvironment. The viewer immersed in the mixed-reality environment may belocally present at the particular physical space correlated andsynchronized with the computer-generated lifelike holographic objectsand the real physical objects in one or more mixed-reality artificiallayers superimposed on the particular physical space. Alternatively, theviewer may also be remotely located in a different physical space butstill correlated and synchronized with the particular physical space tobe immersed in a holographic mixed-reality (HMR) environment thatprovides the computer-generated lifelike holographic objects, whereinthe HMR environment is synthesized and guided in real time through amixed-reality recording headset worn by an onsite surrogate visitor tothe particular physical space. In the alternate embodiment of theinvention, the remotely-located viewer (i.e. a remote visitor) is alsorequired to wear a head-mounted display (HMD) device or at least utilizea mobile electronic device configured to execute a mixed-reality mobileapplication, in order to experience the holographic mixed-reality (HMR)environment streaming from a surrogate visitor HMR live environmentsharing system and/or a real-world object holographic transport andcommunication room system.

Moreover, for the purpose of describing the invention, a term referredto as “surrogate visitor” is defined as a guide, a curator, amixed-reality experience designer, or another person who is physicallyon-site at a physical landmark, such as a museum, a sports stadium, acultural destination, or another tourist destination, to walk around thephysical landmark while wearing or carrying a mixed-reality recordingheadset to create a computer graphics-generated holographicmixed-reality (HMR) environment for remotely-located visitors'virtualized visit to the physical landmark under the real-time guidanceof the surrogate visitor. In a preferred embodiment of the invention,the surrogate visitor and a plurality of remotely-located visitorsutilize a surrogate visitor holographic mixed-reality (HMR) liveenvironment sharing system and/or a real-world object holographictransport and communication room system that enable real-timevirtualized visiting experience to the remotely-located visitors andtwo-way live interactivity among the surrogate visitor and the pluralityof remotely-located visitors by posting and sharing digitized annotationto a particular artifact, an object of interest, or a specified locationwithin the computer graphics-generated HMR space that correlates to thephysical landmark.

In addition, for the purpose of describing the invention, a termreferred to as “remote visitor,” or “remotely-located visitor” isdefined as a virtual “off-site” visitor to a physical landmark via anintermixture of computer-generated graphics, holograms, and physicalobjects. Typically, the physical landmark under the virtual visit by aremote visitor carries some cultural, historical, event-specific, and/orgeographical significance. The remote visitor, by definition, is noton-site at the physical landmark, but is empowered with real-timeholographic visualization of the physical landmark and interactivitywith the surrogate visitor and/or other remote visitors via two-waydigital annotations on MR objects and locations within the real-timeholographic visualization. In the preferred embodiment of the invention,the real-time holographic visualization of the physical landmark isprovided by a surrogate visitor holographic mixed-reality (HMR) liveenvironment sharing system and/or a real-world object holographictransport and communication room system.

Furthermore, for the purpose of describing the invention, a termreferred to as “HoloWalks” is defined as a novel electronic system thatprovides, executes, enables, and manages a three-dimensional (3D)mixed-reality (MR) space with at least one MR artificial layersuperimposed on a physical space, a mixed-reality (MR) experienceconstruction conceived by an MR experience designer (i.e. a userinteraction choreography designer), and a 3D MR experience sharing withtourists, visitors, and other users who visit the physical space whilewearing a head-mounted display device or utilizing an MR-enabled mobileapplication executed on a mobile device.

In addition, for the purpose of describing the invention, a termreferred to as a “mixed-reality artificial layer” is defined as acomputer-generated graphics layer in which mixed-reality objects (MROs)and mixed-reality holographic human guides or curators are created andpositioned by a 3D mixed-reality space and experience constructionsharing system onto virtual coordinates, which correlate to a particularphysical space of a viewer's interest, such as a tourist destination, amuseum, or an exhibition venue.

Moreover, for the purpose of describing the invention, a term referredto as “hologram” is defined as a three-dimensional holographic objectconfigured to be displayed from a head-mounted display (HMD) device, amobile device executing a mixed-reality visual mobile application, oranother electronic device with a visual display unit. Typically, ahologram is capable of being animated as a three-dimensional elementover a defined period of time. Examples of holograms utilizedmixed-reality environments generated by a 3D mixed-reality space andexperience construction sharing system and/or a real-world objectholographic transport and communication room system include, but are notlimited to, a humanized holographic tour guide, a humanized museumcurator, a humanized travel assistant, a mixed-reality object (MRO), oranother mixed-reality hologram, which can be intermixed with orjuxtaposed to physical objects for seamlessly-vivid visualizations ofboth artificial holograms and physical objects.

In addition, for the purpose of describing the invention, a termreferred to as “three-dimensional model,” or “3D model,” is defined asone or more computer-generated three-dimensional images, videos, orholograms. In a preferred embodiment of the invention, a computerized 3Dmodel is created as a hologram after multi-angle video data areextracted, transformed, and reconstructed by three-dimensional graphicsprocessing algorithms executed in a computer system or in a cloudcomputing resource comprising a plurality of networked andparallel-processing computer systems. The computer-generated 3D modelcan then be utilized as a mixed-reality object (MRO) or a humanizedmixed-reality hologram (MRH) in a mixed-reality artificial layersuperimposed on a particular physical space correlated by virtualcoordinates from a 3D mixed-reality space and experience constructionsharing system and/or a real-world object holographic transport andcommunication room system.

Moreover, for the purpose of describing the invention, a term referredto as “cloud” is defined as a scalable data network-connected and/orparallel-processing environment for complex graphics computations,transformations, and processing. The data network-connected and/orparallel-processing environment can be provided using a physicalconnection, a wireless connection, or both. For example, a cloudcomputing resource comprising a first cloud computing server, a secondcloud computing server, and/or any additional number of cloud computingservers can each extract and transform a portion of multi-angle videodata simultaneously as part of a scalable parallel processing algorithm,which performs temporal, spatial, and photometrical calibrations, andexecutes depth map computation, voxel grid reconstruction, and deformedmesh generation. A scalable number of cloud computing servers enables areal-time or near real-time transformation and reconstruction of 3Dmodels after consumer video recording devices transmit multi-angle videodata to the cloud computing resource.

Furthermore, for the purpose of describing the invention, a termreferred to as “HoloPortal” is defined as a 3D model creation studiothat incorporates cameras positioned on a multiple number of anglesaround a stage, where a target object is placed for video footagerecording at the multiple number of angles around the stage. The 3Dmodel creation studio also typically incorporates a real-time or nearreal-time 3D reconstruction electronic system, which is configured toperform silhouette extractions, 3D voxel generation, 3D mesh generation,and texture and detail-adding operations to create a user-controllablethree-dimensional model that resembles the target object.

In addition, for the purpose of describing the invention, a termreferred to as “HoloCloud” is defined as a novel electronic system thatcaptures live multi-angle video feeds of a target object with portableelectronic devices and generates a user-controllable three-dimensionalmodel by performing various 3D reconstruction calculations andprocedures in a scalable cloud computing network. In one example, aHoloCloud system comprises a plurality of common consumer-level videorecording devices (e.g. smartphones, camcorders, digital cameras, etc.)positioned in various angles surrounding a target object (e.g. a human,an animal, a moving object, etc.), a scalable number of graphicprocessing units (GPU's) in a scalable cloud computing platform, a 3Dpre-processing module, a 3D reconstruction module, a background 3Dgraphics content, a 360-degree virtual reality or video content, and adynamic 3D model created by the 3D reconstruction module. In oneembodiment, the 3D pre-processing module and the 3D reconstructionmodules are graphics processing software executed in the scalable numberof graphic processing units (GPU's). In another embodiment, thesemodules may be hard-coded specialized semiconductor chipsets or anotherhardware that operate in conjunction with the GPU's to provide 3Dprocessing and reconstruction.

FIG. 1 shows a process flow (100) for a surrogate visitor holographicmixed-reality (HMR) live environment sharing system withremotely-located visitors, in accordance with an embodiment of theinvention. A first step of operating the surrogate visitor HMR liveenvironment sharing system involves requesting a surrogate visitor towear a mixed-reality (MR) recording headset or carry another MRrecording device, and visit a physical landmark directly, as shown inSTEP 101. Then, the surrogate visitor scans the physical landmark inreal-time with the MR recording device by walking around the physicallandmark, wherein the MR recording device is part of the surrogatevisitor HMR live environment sharing system.

In a preferred embodiment of the invention, the surrogate visitor HMRlive environment sharing system is operatively connected to theHoloWalks system that can synthesize computer-generated holographicspaces based on the walk-through motions visualized by the MR recordingdevice, which is worn or carried by the surrogate visitor at thephysical landmark. As shown in STEP 102, the MR recordingdevice-captured field of vision from the surrogate visitor thenundergoes graphical image processing and transformations in theHoloWalks system to produce holographic space structures and holographicobjects that correspond to the physical landmark. The holographic spacestructures and objects synthesized by the HoloWalks system are thenreal-time streamed to a remote visitor's separate physical space, whichis a different location from the location of the physical landmark, asshown in STEP 103.

If the remote visitor is wearing a head-mounted display (HMD) orutilizes another device capable of visualizing mixed-realityenvironments, the remote visitor is able to see the holographic space ofthe physical landmark and related structures and objects as holograms,which are superimposed to the remote visitor's separate physical space,as shown in STEP 104. This off-site-based virtualized visit to thephysical landmark via the HMD worn in the separate physical space of theremote visitor relieves the physical landmark from overcrowding, on-sitemixed-reality visualization equipment (e.g. HMD) availabilityrequirements, on-site HMD shortage or theft risks, and other logisticalbottlenecks associated with on-site mixed-reality implementations.

Importantly, the surrogate visitor HMR live environment sharing systemcan provide immersive holographic mixed-reality (HMR) environments in ascalable manner to any number of remote visitors, wherein each instanceof the HMR environment provided to each remote visitor corresponds to aunique and individualized HMR instance of the actual physical landmark,with narrations and explanations provided in a live real-time session bythe surrogate visitor, who can appear as a hologram in each instance ofthe HMR environment experienced by each remote visitor. Furthermore, inthe preferred embodiment of the invention, each remote visitor is ableto insert a digital annotation (i.e. containing comments, notes,questions, etc.) to a holographic object or to a particular location inthe HMR representation of the physical landmark, and share the digitalannotation in real time with the surrogate visitor and peer visitors.

FIG. 2 shows a system block diagram (200) for a surrogate visitor HMRlive environment sharing system with remotely-located visitors, inaccordance with an embodiment of the invention. The surrogate visitorHMR live environment sharing system in this embodiment comprises amixed-reality (MR) recording device worn or carried by a surrogatevisitor (201), a holographic mixed-reality (HMR) space scan engine (211)originating from a HoloWalks system and connected to the MR recordingdevice, a surrogate visitor HMR live environment sharing platform (203,205, 207) that includes a HMR space synthesis module (203), a graphicsprocessing server (205), and an HMR space streaming server (207), and ahead-mounted display (HMD) device worn by a remote visitor (209), asshown in FIG. 2.

In the preferred embodiment of the invention, the HMR space scan engine(211) is also part of the HoloWalk system's walk-through map creationengine and 3D map databases (e.g. 707 in FIG. 7), and is capable ofrecognizing and interpreting meaning of a scanned object or structure ofa physical landmark. The intelligent machine recognition andinterpretation of scanned objects and structures during the surrogatevisitor's walk-through stages (i.e. STEPs 101˜102 in FIGS. 1 and 201 inFIG. 2) enable real-time and dynamic digital annotations (i.e.virtualized remotely-located visitor comments, notes, questions,multimedia postings, etc.) to specified holographic structures andobjects targeted by one or more remote visitors to the HMR liveenvironment, which simulates the experience of visiting the physicallandmark in real-time, aided by curation or guide by the surrogatevisitor on-site at the physical landmark.

Continuing with the system block diagram (200) for the surrogate visitorHMR live environment sharing system illustrated in FIG. 2, the surrogatevisitor HMR live environment sharing platform (203, 205, 207)incorporates the HMR space synthesis module (203) that creates 3Dholographic structures and objects after receiving digitizedvisualization data from the mixed-reality (MR) recording device worn orcarried by the surrogate visitor (201). The computer graphics generationof the 3D holographic structures and objects by the HMR space synthesismodule (203) is at least partly executed by the graphics processingserver (205) for computer graphics generation, conversion,transformations, and high-resolution synthesis of various holograms thatcorrespond to the physical landmark and the objects contained in thephysical landmark. The various holograms created by the HMR spacesynthesis module (203) and the graphics processing server (205)constitute the holographic space representing the physical landmark,which is subsequently “teleported” to a remote visitor's physicallocation by HMR space streaming initiated by the HMR space streamingserver (207), as shown in FIG. 2.

In the preferred embodiment of the invention, the surrogate visitor HMRlive environment sharing platform (203, 205, 207) also incorporates anHMR space sharing tool and apps (213) that include a scalable HMR spacestreaming server architecture (215) and an HMR space sharing apps (217)for head-mounted displays (HMDs) worn by a plurality of remote visitors.The scalable HMR space streaming server architecture (215) assigns avariable number of hologram-streaming computer servers, depending on acurrent number of HMR space streaming requests by the plurality ofremote visitors. For instance, if a larger number of remote visitors isrequesting hologram-based virtualized visit to the physical landmark ata given time frame, the scalable HMR space streaming server architecture(215) activates a correspondingly-increased number of HMR spacestreaming servers (207) for real-time live HMR space sharing with thelarger number of remote visitors. Likewise, if a smaller number ofremote visitors is requesting hologram-based virtualized visit to thephysical landmark at a given time frame, the scalable HMR spacestreaming server architecture (215) correspondingly reduces the numberof active HMR space streaming servers (207) for real-time live HMR spacesharing to optimize network traffic and system resource management forthe surrogate visitor HMR live environment sharing system.

Furthermore, in the preferred embodiment of the invention, the HMR spacesharing apps (217) for a head-mounted display (HMD) worn by a remotevisitor are configured to receive, decode, and display the “teleported”holographic mixed-reality (HMR) space as a live stream, which representsthe real-time holographic instance of the physical landmark curatedand/or narrated by the on-site surrogate visitor. Alternatively, the HMRspace sharing apps (217) may be executed by a smart phone or anothermobile device carried by the remote visitor, wherein the HMR spaceenvironment is provided by the display screen of the mobile deviceinstead of the HMD. Preferably, the HMR space sharing apps (217) alsoenable interactive virtualized visiting experience by allowing a remotevisitor to post real-time digital annotations to selected holographicobjects and structures in the HMR space. The posted real-time digitationannotations, which may be the remote visitor's comments, notes,questions, or multimedia information for a particular object orstructure in the HMR space representing the physical landmark, can beshared with the surrogate visitor or other remote or local visitors forinteractive feedback and communications. In the preferred embodiment ofthe invention, the HMR space sharing apps (217) for head-mounteddisplays (HMDs) may be configured to operate in a mobile operatingsystem environment (e.g. Android, iOS, Windows 10 Mobile, etc.) executedby application processors, CPUs, and memory units incorporated inhead-mounted displays (HMDs), smart phones, tablet computers, or othermobile electronic devices utilized by remote visitors.

As illustrated by the system block diagram (200) in FIG. 2, one or moreremote visitors (209) can achieve “remote visiting” of the physicallandmark by immersing into the real-time holographic mixed-reality (HMR)environment with holographic objects and structures that mirror physicalobjects and structures at the physical landmark. The real-time HMRenvironment is typically visualized through a head-mounted display (HMD)that executes the HMR space sharing app (217). In the preferredembodiment of the invention, the real-time HMR environment isexperienced at the comfort of a remote visitor's local space (e.g. home,classroom, or office space), where the physical structures of the remotevisitor's local space may also be visible through the HMD, even if theholographic objects and structures of the real-time HMR environment fromthe physical landmark is displayed in the HMD, as the system is capableof providing a uniquely “mixed-reality” perspective, as opposed to anentirely-synthetic (i.e. virtual reality) perspective. Optionally, theremote visitor or the system operator may choose to visualize thereal-time HMR environment of the physical landmark without an infusionof the physical objects in the remote visitor's physical space.

The surrogate visitor HMR live environment sharing system, asillustrated in FIG. 2, is configured to host various holographicmixed-reality (HMR) remote sharing events (219) that involveinternational real-time HMR space sharing events for cultural orhistorical landmarks (221). For example, a surrogate visitor, who is acertified curator or a tour guide, may physically visit a museum inLondon, and initiate the walk-through space scanning (e.g. STEPs 101˜103in FIG. 1) at a pre-arranged time slot, where a plurality ofremotely-located virtual visitors from New York, San Francisco, Seoul,and Tokyo log into the HMR space streaming server (207) of the surrogatevisitor HMR live environment sharing system through their head-mounteddisplay (HMD) devices, which execute HMR space sharing apps (217) tovisualize the HMR space of the museum in London for a live tour sessionunder the curation of the surrogate visitor. In this example, eachremotely-located virtual visitor to the museum can initiate a digitalannotation to insert notes, comments, questions, or multimediainformation to a particular holographic object or structure displayed inthe HMR space for real-time sharing with the surrogate visitor or peerremote visitors.

FIG. 3 shows an exemplary application diagram (300) for a surrogatevisitor HMR live environment sharing system with remotely-locatedvisitors, in accordance with an embodiment of the invention. In thisapplication example, a surrogate visitor at a museum exhibit (305A)initiates digital scanning of his or her surroundings by wearing orcarrying a mixed-reality (MR) recording device and conducting awalk-through of the museum exhibit, after invoking the HoloWalks system.The HoloWalks system provides a mixed-reality user experience designermenu as a graphical user interface within the field of vision of thesurrogate visitor, and allows the surrogate visitor to implement varioususer interaction sequences and multimedia experience routines within aholographic mixed-reality (HMR) space environment (305B), which can besubsequently teleported to a classroom space (307A) occupied by someremotely-located visitors. The HoloWalks system also operates inconjunction with the surrogate visitor HMR live environment sharingplatform (e.g. 203, 205, 207 in FIG. 2) that synthesizes HMR structuresand objects for real-time HMR space streaming to remote visitors.

Once the teleportation of the HMR environment that represents the museumexhibit is activated as a live 3D “space streaming” event, then the HMRspace environment (305C) simulated by computer graphics is superimposedto a portion (307B) of the classroom space (307A), wherein the HMR spaceenvironment (305C) is only visible by remote visitors through HMDs orother mobile device displays that execute HMR space sharing apps (e.g.217 in FIG. 2). The remote visitors in the classroom space (307B) inthis example are able to experience two-way live interactions with themuseum exhibit, the surrogate visitor, and other on-site or remotevisitors by posting comments, notes, multimedia notes, and questions asdigital annotations to a specific holographic object or structureperceived in the HMR space environment (305C).

The exemplary application diagram (300) for the surrogate visitor HMRlive environment sharing system in FIG. 3 also includes remote one-wayviewers (301, 303), who can watch the museum exhibit and the surrogatevisitor's guided tours as live or recorded playback events. For example,one remote one-way viewer (301) utilizes a notebook computer and asocial-media video streaming portal to watch the museum exhibit on atwo-dimensional computer display screen, while another one-way viewer(303) utilizes a head-mounted display device that presents the museumexhibit visualization as a pure virtual reality (VR) environment, whichlacks bearing and coordination with physical objects in the viewer's ownspace.

FIG. 4 shows an example (400) of mixed-reality space mapping formatcompatibility-enhancing dual-file structures for a 3D mixed-realityspace and experience construction sharing system and a surrogate visitorHMR live environment sharing system, in accordance with an embodiment ofthe invention. As shown by this example (400), a native 3D map file(401) may have been initially generated by a mixed-reality experiencedesigner or a surrogate visitor utilizing a head-mounted display deviceor another portable electronic device connected to a HoloWalks creatormodule and a walk-through map creation engine. The newly-created native3D map file (401) is stored in a 3D mixed-reality map database (413)executed in a memory and a CPU of a 3D map graphics-processing computerserver (411).

Furthermore, in this embodiment of the invention that implements themixed-reality space mapping format compatibility-enhancing dual-filestructures, the newly-created native 3D map file (401) alsoautomatically undergoes an XML-compliant 3D map format encoding via anXML-based 3D map format encoder (403), as shown in FIG. 4. The XML-based3D map format encoder (403) first fetches mapped space information,location information, geotagging information, timestamps, map scanningdevice information, and/or map creation user information from the native3D map file (401) using a mixed-reality space mapping formatcompatibility-enhancing data fetcher (405). Then, using a particularmixed-reality 3D map standardization scheme, an XML file synthesizer(407) generates an XML-based 3D map format structure that can flexiblybe converted into a proprietary 3D map format on the fly, depending onthe graphics format compatibilities provided by a 3D map encoding deviceutilized by a mixed-reality environment viewer. The XML-based 3D mapformat structure generated from the XML file synthesizer (407) is thensaved as an XML file (409), which is subsequently transmitted to the 3Dmixed-reality map database (413) and stored and paired with the native3D map file (401) as the dual-file structure for that particular 3D map.

Then, when the mixed-reality environment viewer selects that particular3D map to invoke a mixed-reality environment using a 3D map decodingdevice (419) (e.g. a viewer's HMD, a viewer's smart phone, or anotherportable electronic device executing a mixed-reality visualization app),a 3D map file compatibility detector module (415) inspects anddetermines whether the 3D map decoding device (419) is compatible withthe native 3D map file (401). On the one hand, if the 3D map decodingdevice (419) is determined to decode the native 3D map file (401)without any mixed-reality map data format compatibility problems, the 3Dmap file compatibility detector module (415) relays the native 3D mapfile (401) to the 3D map decoding device (419). On the other hand, ifthe 3D map decoding device (419) is deemed incompatible with decodingthe native 3D map file (401), then a compatible 3D map format creationmodule (417) creates a 3D map decoding device-compatible format on thefly (i.e. dynamically) from the XML file (409) stored in the 3Dmixed-reality map database. Because the on-the-fly map file conversionsare often merely best machine-determined presumptions of the format(s)likely decodable by the 3D map decoding device (419), the compatible 3Dmap format creation module (417) may send a warning notice to the 3D mapdecoding device (419) that some mixed-reality objects and mixed-realitymaps may not be perfect visual representations of what the mixed-realityexperience designer conceived at the mixed-reality environment scenariocreation stage.

FIG. 5 shows a mixed-reality space map creation and mapping formatcompatibility-enhancing method flowchart (500) for a 3D mixed-realityspace and experience construction sharing system and a surrogate visitorHMR live environment sharing system, in accordance with an embodiment ofthe invention. In the first step of the mixed-reality space map creationand mapping format compatibility-enhancing method flowchart (500), the3D mixed-reality space and experience construction sharing system (i.e.the HoloWalks system) registers a native 3D map file in a 3Dmixed-reality map database, as shown in STEP 501. Then, the HoloWalkssystem fetches mapped space information, location information,geotagging information, timestamps, map scanning device information, andmap creation user information from the native 3D map file, as shown inSTEP 502.

By utilizing an XML-based 3D map format encoder, as previously describedin conjunction with FIG. 4, the fetched information is then arranged andstructured into an XML-compliant file in accordance with a particularmixed-reality 3D map standardization scheme, as shown in STEP 503. Then,both the native 3D map file and the corresponding XML-compliant file arestored as a paired data structure in the 3D mixed-reality map database,as shown in STEP 504.

When a mixed-reality experience viewer's 3D map decoding device invokesgraphical rendering of a mixed-reality environment represented by thenative 3D map file, the HoloWalks system checks and determines whetherthe mixed-reality experience viewer's 3D map decoding device supportsgraphical decoding compatibility with the native 3D map file, as shownin STEP 505. If the native 3D map file is indeed supported by themixed-reality experience viewer's 3D map decoding device, then theHoloWalks system transmits the native 3D map file to the mixed-realityexperience viewer's 3D map decoding device for perfectly-accuratemixed-reality environment graphical rendering, as shown in STEP 508. Onthe other hand, if the native 3D map file is not supported by themixed-reality experience viewer's 3D map decoding device, then theHoloWalks system parses the XML-compliant file correlated to the native3D map file, and dynamically creates a new 3D map format presumed to becompatible with the mixed-reality experience viewer's 3D map decodingdevice, as shown in STEP 506. The new 3D map format is then transmittedto the mixed-reality experience viewer's 3D map decoding device, alongwith a warning stating that some mixed-reality environment contents maynot appear perfect due to potential graphical format compatibilityshortcomings, as shown in STEP 507.

FIG. 6 shows a process flow diagram (600) for a three-dimensional (3D)mixed-reality space and experience construction sharing system (i.e. the“HoloWalks” system), in accordance with an embodiment of the invention.As illustrated by the process flow diagram (600), the first step in the3D mixed-reality space and experience construction sharing for theHoloWalks system involves creation of a computer graphics-generatedholographic tour guide, or a “hologram-based virtual guide” as labeledin FIG. 6, from a physical person (i.e. a real human tour guide, a realhuman museum curator, or another real human presenter).

In a preferred embodiment of the invention, this holographic tour guideis created from a HoloPortal studio, a HoloCloud system, or a real-worldobject holographic transport and communication room system, which areconfigured to capture, synthesize, and transform various real-lifeobjects and humanized figures into holographically-displayable datasetsthat can be utilized in creation of various mixed-reality objects (MROs)and mixed-reality holograms (MRHs) that can be subsequently intermixedwith or positioned next to physical objects in a partiallycomputer-generated mixed-reality environment, when viewed from ahead-mounted display or another mixed-reality viewing-enabled portabledisplay device in a particular physical space of a popular traveldestination by a tourist, a visitor, or another user. The holographictour guide is typically an animated 3D hologram over an experiencedesigner-defined timeframe, which is called herein as a “4D” (i.e.four-dimensional) holographic content in the HoloWalks system to accountfor the extra dimension of holographic animation time, in addition to x,y, and z coordinates comprising the three dimensions of the 3D hologramitself.

In one example, the holographic tour guide creation is computergraphics-generated and executed by a holographic image capture studio,such as the HoloPortal system (e.g. 701 in FIG. 7) or the real-worldobject holographic transport and communication room system (e.g. FIGS.11-17), which is configured to capture a plurality of multi-angle imagesof a real-life human model (e.g. a “physical guide” in FIG. 6) andgenerate a digitized holographic model content (e.g. a “hologram-basedvirtual guide), as shown in FIGS. 6 and 7. In the preferred embodimentof the invention, the creation of the digitized holographic modelcontent involves eight or more time-synchronized multiple angle imagecaptures of a targeted object, such as the real-life human model oranother real-life object undergoing 3D hologram transformations, tobecome a computer graphics-based mixed-reality object (MRO) or amixed-reality hologram (MRH). The HoloPortal component (e.g. 701 in FIG.7) in the Holowalks system then executes a volumetric conversion of theeight or more time-synchronized multiple angle image captures to createthe digitized holographic model content, which can be controlled andchoreographed as a hologram by a mixed-reality experience designer oranother user interaction choreography director that utilizes a HoloWalkscreator module (e.g. 709 in FIG. 7) connected to a walk-through mapcreation engine and 3D map database (e.g. 707 in FIG. 7) and a userexperience choreography engine and 3D holographic database (e.g. 705 inFIG. 7) in the 3D mixed-reality space and experience constructionsharing system (i.e. the HoloWalks system), as shown in FIGS. 6 and 7.

Preferably, the HoloPortal (e.g. 701 in FIG. 7) or the real-world objectholographic transport and communication room system (e.g. FIGS. 11-17)is a component of the 3D mixed-reality space and experience constructionsharing system (i.e. the HoloWalks system). As shown in FIG. 6, theHoloWalks system incorporates a mixed-reality (MR) guided tourapplication development process, which involves creating and managing ahologram model database and mixed-reality (MR) applications thatincorporate holograms and virtual reality plug-in components. Thevirtual reality plug-in components in MR applications, with userexperience designer-selected hologram models and objects, enables 3Dmixed-reality space and experience construction to generate agraphically-immersive and interactive mixed-reality environment formixed-reality guided tour content creations, which are typicallysuperimposed as a mixed-reality artificial layer on a particularphysical location (e.g. a popular tourist spot, a museum exhibit, etc.).

In the preferred embodiment of the invention, the MR guided tourapplication development process, as shown in the process flow diagram(600) in FIG. 6, also involves proprietary or standardized holographicdata compression of the digitized holographic model content (e.g.holographic tour guides, holographic museum curators, MROs, MRHs, etc.).In one example, the digitized holographic model content can becompressed and sub-divided as an object (OBJ) file and a digital assetexchange (DAE) file, wherein the OBJ file contains compressedmulti-angle graphics data representing a particular holographic model,and wherein the DAE file contains digital graphics compatibility schemesand/or information, typically based on COLLADA (collaborative designactivity) XML schema or another industry-standardized graphicscompatibility scheme.

Then, the compressed holographic model content files (e.g. OBJ and DAEfiles) can be utilized by one or more holographic app and servicetoolkits, such as WebGL, Unity, and Unreal Engine, by HoloWalks contentcreators/mixed-reality experience designers to envision, generate,modify, and manage a variety of HoloWalks applications and serviceofferings. In one embodiment, the holographic app and service toolkitsmay be integrated into or operatively connected to a user experiencechoreography engine and a walk-through map creation engine in aHoloWalks Cloud module (e.g. 703 in FIG. 7 in the HoloWalks system. Inanother embodiment, the holographic app and service toolkits may also bepartially incorporated into a holographic studio/room component (e.g.701 in FIG. 7) and/or a HoloWalks Creator module (e.g. 709 in FIG. 7),wherein the HoloWalks Creator module utilizes the user experiencechoreography engine and the walk-through map creation engine in theHoloWalks Cloud module to provide an MR experience constructioninterface (e.g. FIGS. 8A˜8C) to an MR experience designer to enableintelligent machine-generated 3D map via user walk-through andchoreographed MR guided tour content creation, as illustrated in theprocess flow diagram (600) in FIG. 6 and further elaborated in FIG. 7,FIGS. 8A˜8C, and FIGS. 10C˜10H.

FIG. 7 shows a system block diagram (700) for a three-dimensional (3D)mixed-reality space and experience construction sharing system (i.e. the“HoloWalks” system), in accordance with an embodiment of the invention.In a preferred embodiment of the invention, the HoloWalks systemincludes a dedicated holographic image capture studio (i.e. a HoloPortal(701)) or a real-world object holographic transport and communicationroom component (e.g. FIGS. 11-17). The HoloWalks system may contain aplurality of HoloPortal studios or the real-world object holographictransport and communication room components (e.g. FIGS. 11-17) that arephysically constructed, activated, and connected to the HoloWalks systemfrom various locations around the world, as illustrated by variousholographic studios or rooms in London, Tokyo, Seoul, and otherlocations in FIG. 7. In the preferred embodiment, this dedicatedholographic image capture studio incorporates cameras positioned on amultiple number of angles around a stage, where a target object isplaced for video footage recording at the multiple number of anglesaround the stage. This specialized studio also typically connects themulti-angle cameras around the stage to a real-time or near real-time 3Dreconstruction electronic system comprising 3D reconstruction computergraphics software, which is executed on a CPU and a memory unit of ahigh-performance computer server suited for intensive graphicsprocessing. The real-time or near real-time 3D reconstruction electronicsystem in the HoloPortal (701) is configured to perform silhouetteextractions, 3D voxel generation, 3D mesh generation, and texture anddetail-adding operations to create a user-controllable three-dimensionalmodel that resembles the target object.

The HoloWalks system also includes a HoloWalks Cloud module (703), whichincorporates a user experience choreography engine and a 3D holographicdatabase (705) and a walk-through map creation engine and 3D mapdatabase (707), as shown in the system block diagram (700) in FIG. 7.The user experience choreography engine in the HoloWalks Cloud module(703) allows an MR experience designer to select a holographic objectfrom the 3D holographic database and place the holographic object in aparticular location of a 3D map comprising one or more MR artificiallayers and virtual coordinates superimposed on a physical space.Furthermore, the user experience choreography engine also allows the MRexperience designer to create a series of prespecified or potentialinteractions, or “choreographies,” between the holographic object and aHoloWalks viewer. The holographic object may be a holographic humanizedtour guide, a holographic museum curator, a holographic mixed-realityobject (MRO), a mixed-reality hologram (MRH), or another hologram thatcan be placed into a mixed-reality environment with one or more MRartificial layers and virtual coordinates superimposed on a physicalspace, which is typically a popular tourist spot, a museum exhibit, anevent venue, or another tourist attraction.

Furthermore, the walk-through map creation engine and 3D map database(707) in the HoloWalks Cloud module (703) enables the MR experiencedesigner to wear a head-mounted display (HMD) device and walk-through atargeted physical space (e.g. a particular location in a museum, apopular tourist spot, etc.), which activates the walk-through mapcreation engine in the HoloWalks Cloud module (703) to intelligently andautomatically generate a 3D map from visual information gathered by theHMD. This intelligent machine-generated 3D map can be utilized as amixed-reality artificial layer with virtual coordinates superimposed onthe targeted physical space, and stored in the 3D map database in theHoloWalks Cloud module (703).

In the preferred embodiment, the HoloWalks Cloud module (703) is a 3Dgraphics-generating software element for the walk-through map creationengine and for the user experience choreography engine, as shown in thesystem block diagram (700) in FIG. 7. The HoloWalks Cloud module (703)also incorporates or connects to 3D map and 3D holographic relationaldatabases as dynamic storages of 3D maps and holograms generated by the3D graphics-generating software element. Typically, the HoloWalks Cloudmodule (703) is executed by a scalable number of CPUs, GPUs, and memoryunits in one or more high-performance cloud-networked computer serverssuited for 3D graphics processing. The HoloWalks Cloud module (703) isalso operatively connected to the HoloPortal component (701), theHoloWalks Creator module (709), authorized MR experience designer HMDs,smart phones, notebook computers, and personal computers, as shown inthe system block diagram (700) in FIG. 7.

Furthermore, in the preferred embodiment of the invention, the MRexperience designer wears a head-mounted display (HMD) device (713) orutilizes a portable electronic device connected to the HoloWalks Creatormodule (709) to create a 3D map on the fly at the vicinity of thetargeted physical space or to select a stored 3D map from the 3D mapdatabase, and positions one or more holographic objects stored in the 3Dholographic database at designer-desired specific virtual coordinates ofa mixed-reality artificial layer relative to the targeted physical spaceby invoking a MR user experience construction interface. Preferably, theMR user experience construction interface provides intuitive gesturecommands and user experience design choreography construction andcontrols on a partially-transparent menu visible in the HMD or inanother portable electronic device, as illustrated in FIGS. 8A˜8C andFIGS. 10C˜10H.

In one embodiment of the invention, the digitized holographic modelcontent stored in the 3D holographic database in the HoloWalks Cloudmodule (703) can be compressed and sub-divided as an object (OBJ) fileand a digital asset exchange (DAE) file, wherein the OBJ file containscompressed multi-angle graphics data representing a particularholographic model, and wherein the DAE file contains digital graphicscompatibility schemes and/or information, typically based on COLLADA(collaborative design activity) XML schema or anotherindustry-standardized graphics compatibility scheme.

Then, the compressed holographic model content files (e.g. OBJ and DAEfiles) can be utilized by one or more holographic app and servicetoolkits, such as WebGL, Unity, and Unreal Engine, by HoloWalks contentcreators/mixed-reality experience designers to envision, generate,modify, and manage a variety of HoloWalks applications and serviceofferings. In context of the system block diagram (700) in FIG. 7, theholographic app and service toolkits may be integrated into oroperatively connected to the user experience choreography engine and thewalk-through map creation engine in the HoloWalks Cloud module (703) inthe HoloWalks system. Alternatively, the holographic app and servicetoolkits may also be partially incorporated into the HoloPortalcomponent (701) and/or the HoloWalks Creator module (709). The HoloWalksCreator module (709) utilizes the user experience choreography engineand the walk-through map creation engine in the HoloWalks Cloud module(703) to provide an MR user experience construction interface (e.g.FIGS. 8A˜8C) to an MR experience designer, wherein the MR userexperience construction interface is configured to create an intelligentmachine-generated 3D map simply by user walk-through and to generate andchoreograph MR guided tour contents with holographic objectssuperimposed on a targeted physical space containing physical objects,as demonstrated in FIGS. 8A˜8C and FIGS. 10C˜10H.

The system block diagram (700) for the HoloWalks system in FIG. 7 alsoincorporates a HoloWalks Viewer module (711), which comprises one ormore software sub-modules executed in a CPU, a GPU, and/or a memory unitof a viewer's head-mounted display device (715, 717) or another portableelectronic device (e.g. a smart phone, a table computer, etc.) capableof displaying mixed-reality environments via camera-enabled mobileapplications. The HoloWalks Viewer module (711) incorporates a viewer'smixed-reality visualization interface executed and displayed by one ormore portable electronic devices utilized by the mixed-realityexperience viewer. Furthermore, as shown in FIG. 7, the HoloWalks Viewermodule (711) is configured to download and playback a plurality of“HoloWalks Events” choreographed and implemented by one or more MRexperience designers that utilized one or more HoloWalks Creator modules(709) to generate and insert 3D maps, holographic object positioning,holographic object and physical object interactions and choreographies,and informative historic, cultural, and/or tourism-related details inthe plurality of HoloWalks Events. In context of various embodiments ofthe present invention, HoloWalks Events are immersive mixed-realityscenarios that are executed and displayed in the viewer's head-mounteddisplay device (715, 717) or another portable electronic deviceoperatively connected to the HoloWalks Viewer module (711) and/or therest of the components (i.e. 709, 703, 701) in the HoloWalks system.

Furthermore, in one embodiment of the invention, the HoloWalks systemmay also incorporate a holographic mixed-reality browser and athird-party holographic application loader that are configured to loadand display various holographic third-party HoloWalks apps by connectingto and downloading various software applications from a cloud-connectedcomputer server, which executes a third-party holographic applicationdatabase and a tourist or museum curation-related hologram softwaredevelopment kit (e.g. HoloWalks SDK) for implementation and deploymentof various tourist or visitor-assisting holographic applications thatcan be utilized in HoloWalks mixed-reality experience environments. Inone instance, “third-party” refers to an independent group of HoloWalksapp developers who are not operated or owned by HoloWalks systemoperators, tourism organizations, or museum owners. In another instance,“third-party” may include tourism counsel-partnered or museumowner-affiliated independent app developers.

FIG. 8A shows a HoloWalks experience designer walking through a physicalspace (800A), which enables the HoloWalks system to automatically andintelligently create three-dimensional (3D) digitized mappingvisualization data for synthesizing a mixed-reality artificial layerconstruction, in accordance with an embodiment of the invention. In apreferred embodiment of the invention, the HoloWalks experience designeris type of a mixed-reality (MR) experience designer who wears ahead-mounted display (HMD) device and walks through the physical space(800A). A HoloWalks Creator module (e.g. 709 in FIG. 7) connected to awalk-through map creation engine with a 3D map database (e.g. 707 inFIG. 7) and a user experience choreography engine with a 3D holographicdatabase (e.g. 705 in FIG. 7) provides the intelligent machine-generated3D mapping via the MR experience designer walk-through, as visualsensors incorporated in the HMD device scan the vicinity of the physicalspace (800A) and transmit raw imaging data to the HoloWalks Creatormodule and the walk-through map creation engine. Once the walk-throughmap creation engine completes an automated 3D map construction, thenewly-created 3D map can be stored and categorized in the 3D mapdatabase for convenient on-demand retrieval for subsequent mixed-realityuser experience design creations for the physical space (800A).

FIG. 8B shows the MR experience designer selecting a desired spot withinthe three-dimensional (3D) map in a mixed-reality (MR) experienceconstruction interface (800B) to initiate a mixed-reality holographicguide content creation, in accordance with an embodiment of theinvention. Furthermore, the MR experience designer subsequently placesholographic contents and overlays user interaction elements (800C) inthe desired spot within the 3D map by utilizing the mixed-reality (MR)experience construction interface in the HoloWalks system, as shown inFIG. 8C. In this particular example, the MR experience constructiondesigner is placing a holographic museum curator in front of anexhibited painting and choreographing the holographic museum curator'smovements, actions, and/or narrations, so that the future visitors tothe museum can experience the holographic museum curator'sinteractivities in front of the exhibited painting, if the visitors areimmersed in a related mixed-reality environment provided by an HMDdevice or another portable electronic device enabling the mixed-realityenvironment.

In some embodiments of the invention, the holographic objects, such asthe holographic museum curator as shown in FIGS. 8C and 8D, mayintegrate artificial intelligence to answer questions and/or conversedynamically and informatively, if the visitors or tourists ask questionsto the holographic objects while utilizing their HMD devices or othermixed-reality environment-enabling portable electronic devices.Furthermore, the holographic objects and/or contents placed in thedesired spot and the overlayered user interaction elements can betime-sequenced and choreographed relative to potential user interactionswith the holographic and/or physical objects, which can be experiencedin the desired spot through an immersive mixed-reality graphicsenvironment provided by HMD devices or other portable electronic devicesutilized by tourists or museum visitors, when the fully-constructed MRexperience event is activated in the desired spot.

In the usage example sequence as illustrated in FIGS. 8B and 8C, theHoloWalks Creator module (e.g. 709 in FIG. 7) in the 3D mixed-realityspace and experience construction sharing system (i.e. the HoloWalkssystem) utilizes the user experience choreography engine and thewalk-through map creation engine in the HoloWalks Cloud module (e.g. 703in FIG. 7) to provide the MR experience construction interface to the MRexperience designer to enable intelligent machine-generated 3D map viauser walk-through and choreographed MR guided tour content creation.Moreover, the MR experience designer can wear a head-mounted display(HMD) device or utilize a portable electronic device connected to theHoloWalks Creator module to create a 3D map on the fly at the vicinityof the targeted physical space or to select a stored 3D map from the 3Dmap database, and position one or more holographic objects stored in the3D holographic database at designer-desired specific virtual coordinatesof a mixed-reality artificial layer relative to the targeted physicalspace by invoking a MR user experience construction interface.Preferably, the MR user experience construction interface providesintuitive gesture commands and user experience design choreographyconstruction and controls on a partially-transparent menu visible in theHMD or in another portable electronic device, as demonstrated in FIGS.8A˜8C.

FIG. 8D shows a user (800D) interacting with a holographic guide and/orcontents in a mixed-reality environment provided by a HoloWalks viewer,in accordance with an embodiment of the invention. In the preferredembodiment of the invention, the user (800D) is a tourist or a visitorto a museum exhibit, who is immersed in a mixed-reality environmentprovided by an HMD device or another portable electronic device toexperience the holographic museum curator's interactivities in front ofthe exhibited painting, as shown in FIG. 8D. The user (800D) can listento verbal narration and observe animated movements from the holographicmuseum curator, while observing real physical paintings in the museumexhibit in the mixed-reality environment. In some cases, the user (800D)can also ask questions and converse interactively with the holographicmuseum curator, which incorporates an advanced level of artificialintelligence to retrieve historical, cultural, and artistic backgroundinformation from a fine arts database and to synthesize naturalizedspeeches for coherent and dynamic conversations with the user (800D).

FIG. 9 shows an example (900) of multiple mixed-reality (MR) artificiallayers superimposed on a physical space for construction of amixed-reality (MR) application from the three-dimensional (3D)mixed-reality space and experience construction sharing system (i.e. the“HoloWalks” system), in accordance with an embodiment of the invention.Each MR artificial layer is a computer-generated graphics layer in whichmixed-reality objects (MROs) and mixed-reality holographic (MRH) humanguides or curators are created and positioned by the HoloWalks systemonto the MR artificial layer's virtual coordinates that correlate to thephysical space of a potential viewer's interest, such as a touristdestination, a museum, or an exhibition venue.

As shown by the example (900) in FIG. 9, several MR artificial layers(i.e. MR Space 1, MR Space 2, . . . , MR Space N) may be superimposed ontop of the physical space for sophisticated MRO and MRH placements andintricate choreographic interaction experience designs with thepotential viewer. In a preferred embodiment of the invention, the viewerimmersed in the mixed-reality environment is typically required to bepresent at the particular physical space correlated and synchronizedwith the computer-generated lifelike holographic objects and the realphysical objects in one or more mixed-reality artificial layerssuperimposed on the particular physical space. Furthermore, the vieweris also required to wear a head-mounted display (HMD) device or at leastutilize a mobile electronic device configured to execute a mixed-realitymobile application, in order to experience the mixed-reality environmentgenerated by the HoloWalks system.

FIG. 10A shows a first step (1000A) in a mixed-reality (MR) applicationexample, in which a user (e.g. a tourist or a visitor) perceivesphysical objects in a physical space, in accordance with an embodimentof the invention. Because the user does not wear a HMD device orutilizes another portable electronic device to invoke and visualize anMR environment, the user only perceives physical objects present in thephysical space.

FIG. 10B shows a second step (1000B) in the mixed-reality (MR)application example, in which a mixed-reality (MR) experience designerwears a head-mounted display (HMD) in the same physical space toinitiate an MR content creation via an MR experience constructioninterface, in accordance with an embodiment of the invention. In anotherembodiment of the invention, the MR experience designer may insteadutilize a smart phone or a table computer to activate the MR experienceconstruction interface.

FIG. 10C shows a third step (1000C) in the mixed-reality (MR)application example, in which the HoloWalks system enables automatedintelligent three-dimensional (3D) mapping of the physical space via HMDspace scanning by the MR experience designer, in accordance with anembodiment of the invention. As shown in the third step (1000C), the MRexperience designer systematically glances at the entire physical spacewhile wearing the HMD device, which enables the walk-through mapcreation engine in the HoloWalks system to autonomously andintelligently construct a machine-generated 3D map of the physicalspace, as previously described in conjunction with FIGS. 6, 7, and 8A.

FIG. 10D shows a fourth step (1000D) in the mixed-reality (MR)application example, in which the HoloWalks system completes userglare-invoked intelligent 3D mapping of the physical space, inaccordance with an embodiment of the invention. The machine-generated 3Dmap of the physical space is then stored and categorized for dynamicon-demand retrieval in the 3D map database, which is incorporated in theHoloWalks Cloud module, as previously described in conjunction with FIG.7.

FIG. 10E shows a fifth step (1000E) in the mixed-reality (MR)application example, in which the HoloWalks system creates virtualcoordinates on mixed-reality (MR) artificial layer(s) in preparation ofthe MR experience designer's MR content synthesis, in accordance with anembodiment of the invention. As previously shown and described inconjunction with FIG. 9, a multiple number of MR artificial layers canbe superimposed on the physical space, wherein each MR artificial layercontains its own set of virtual coordinates where the MR experiencedesigner can place mixed-reality objects (MROs) and/or mixed-realityholograms (MRHs) and implement choreographic user interactivityelements, artificial intelligence for dynamic conversation capabilitiesassociated with MROs or MRHs, and descriptive information that can bepresented by the MROs or MRHs.

FIG. 10F shows a sixth step (1000F) in the mixed-reality (MR)application example, in which the MR experience designer selects anddirects mixed-reality objects (MROs) and interactions in the MRartificial layer(s) intertwined with physical objects and physicalspace, in accordance with an embodiment of the invention. As shown inthe sixth step (1000F), the MR experience designer can utilize gesturecommands, voice commands, or other action-invoking commands to select,locate, resize, and modify MROs and their choreographic userinteractivity elements, using the MR experience construction interfacedisplayed on the HMD or another portable electronic device.

Furthermore, FIG. 10G shows a seventh step (1000G) in the mixed-reality(MR) application example, in which the MR experience designer places anddirects more MROs and interactions in the MR artificial layer(s)intertwined with physical objects and physical space, in accordance withan embodiment of the invention. Likewise, FIG. 10H shows an eighth step(1000H) in the mixed-reality (MR) application example, in which the MRexperience designer places more MROs, mixed-reality holograms (MRHs),and interactions in the MR artificial layer(s) intertwined with physicalobjects and physical space, in accordance with an embodiment of theinvention.

In the preferred embodiment of the invention, the MR experienceconstruction interface is downloaded or transmitted from the HoloWalksCreator module to the HMD or another portable electronic device utilizedby the MR experience designer. In another embodiment of the invention,the MR experience construction interface can be locally pre-loaded tothe HMD or another portable electronic device, and is executable onCPUs, GPUs, and/or memory units in such devices, even if they are notcurrently connected to the rest of the HoloWalks system at the time ofMR experience construction at the physical space.

Once the MR experience construction is completed, the 3D map data andthe MR user experience choreography data, which are key components ofthe MR experience construction for the physical space, are categorizedand stored in the 3D map database and the 3D holographic database. Insome embodiments of the invention, a dedicated “MR user experience”database may store the 3D map data and the MR user experiencechoreography data separately for each MR user experience scenariocreated within the HoloWalks system.

FIG. 10I shows a ninth step (10001) in the mixed-reality (MR)application example, in which an MR experience viewer equipped with aviewer's HMD engages in lifelike intertwined visualization of MROs,MRHs, and physical objects in the same physical space, when the MR userexperience scenario is executed on the viewer's HMD, if the MRexperience viewer is present at the physical space. The MR experienceviewer may utilize a smart phone, a tablet computer, or another portableelectronic device with a display unit, instead of the HMD, to experiencethe MR user experience scenario, if such portable electronic devices arecapable of executing mixed-reality viewer mobile applications for theHoloWalks system.

FIG. 10J shows a tenth step (1000J) in the mixed-reality (MR)application example, in which the MR experience viewer, while notwearing the HMD, only sees physical objects without visual recognitionof the MROs and the MRHs implemented in the same physical space in theMR artificial layer(s), in accordance with an embodiment of theinvention. The MR experience viewer may be a tourist or a visitor to apopular tourist attraction or an exhibition venue. In some embodimentsof the invention, the MR experience viewer may decide to intermittentlyexperience the mixed-reality environment for rich holographicinteractivity and media information on a case-by-case basis, whileenjoying observation of pure physical objects only in other instances,during his or her visit to the physical space.

FIG. 11 shows a single camera and machine learning-based holographicimage capture example (1100), in accordance with an embodiment of theinvention. In this embodiment, a single red-green-blue (RGB)-depth(RGB-D) camera is utilized to capture the motion of a dynamic target,which is required to rotate around the RGB-D camera, instead ofcapturing three-dimensional volume of the dynamic target conventionallywith a plurality of multi-angle cameras positioned around the dynamictarget. Examples of the dynamic target may include, but are not limitedto, a human subject, an animal, or another movable object targeted for acorresponding holographic representation.

As illustrated in a first view (1101) in this example (1100), the singleRGB-D camera is configured to capture the three-dimensional (3D) volumeof a targeted person (i.e. a dynamic target subject), who is required torotate around the single RGB-D camera. Then, as further illustrated in asecond view (1103) in this example (1103), the captured 3D volume of thedynamic target subject is further refined (i.e. sharpened and improvedin image clarity and resolution), extrapolated, and synthesized with oneor more holographic object reconstruction methods and/or algorithmsexecuted on a graphics server to produce a hologram that replicates the3D appearance of the dynamic target subject. For example, the captured3D volume of the dynamic target subject undergoes computerized graphicaltransformations, such as relighting, subject depth calculations,geometrical extrapolations, and volumetric reconstructions in one ormore machine-learning graphical servers. The resulting hologram producedin this fashion embodies at least some movements or changes in theappearance of the dynamic target subject over a specified amount oftime, thus incorporating an additional dimension of time to function asa 4-dimensional (4D) hologram, wherein the first 3 dimensions arerelated to the 3D volume (i.e. x, y, z coordinates) of the dynamictarget subject, which is synthetically animated over time as the fourthdimension to the hologram.

FIG. 12 shows a 4D dynamic hologram production sequence (1200) involvinga target object-initiated self-rotation around a single camera and amachine-learning apparatus for hologram generation, in accordance withan embodiment of the invention. As shown in a first step (1201) of the4D dynamic hologram production sequence (1200), the dynamic targetsubject, which in this case is a human subject, initiates a 360-degreeself-rotation in front of a single RGB-D camera. Instead of theconventional multi-angle positioning of a plurality of cameras that areutilized in conventional hologram production, this embodiment of thepresent invention requires only one specialty camera designed to capturethe color (i.e. “RGB”) data as well as the subject-depth (D) data fromthe singular position of only one camera, as the dynamic target subjectvoluntarily provides a 360-degree rotation motion in front of the singleRGB-D camera, as shown in the first step (1201).

Once the RGB data and the subject-depth data associated with the dynamictarget subject are captured by the single RGB-D camera, themachine-learning apparatus comprising one or more graphical processingunits (GPUs) integrated in a computer server executes a feedbackloop-based real-time full 4D dynamic reconstruction process that firstcreates a volumetric 4D graphical representation of the dynamic targetsubject from the RGB data and the subject-depth data gathered from thesubject's 360-degree rotating motions. Then, as shown in a second step(1203) of the 4D dynamic hologram production sequence (1200), theinitial version of the volumetric 4D graphical representation undergoesintelligent machine-learning with real-time subject RGB and depth dataparameters and artificial intelligence-based graphical extrapolationsand estimations in a continuous feedback loop to produce more refined(i.e. sharpened, improved) geometries for the volumetric 4D graphicalrepresentation of the dynamic target subject. After multiple rounds ofthe intelligent machine-learning refinements of the volumetric 4Dgraphical representation of the dynamic target subject in the continuousfeedback loop, the resulting volumetric 4D graphical representationbecomes a higher-resolution volumetric hologram that sufficientlysatisfies user needs for multi-party holographic applications andcommunications in a mixed-reality (MR) environment.

FIG. 13 shows a cost and convenience advantage example (1300) of thesingle RGB-D camera and machine learning-based holographic image capturemethod (1303) relative to a conventional hologram-generating method(1301) that involves conventional multi-angle position cameras andrelated conventional graphics processing equipment, in accordance withan embodiment of the invention.

As shown in this figure, the conventional hologram-generating method(1301) typically utilizes conventional digital cameras, which lack theability to provide depth (D) parameters of a target subject with asingle angle alone. In the conventional hologram-generating method(1301), a plurality of conventional digital cameras are placed aroundthe target subject and the angle of each camera is carefullypre-calibrated and inputted into one or more conventional (i.e. nonmachine-learning based) graphics servers to construct a hologram.Therefore, the logistics of setting up the pre-calibrated angularpositions of multiple cameras are often overly complicated, and theexorbitant expense of utilizing multiple cameras and multipleconventional graphics servers acts as a barrier to a widespread adoptionof hologram generation and holographic communications.

In contrast, the single RGB-D camera and machine learning-basedholographic image capture method (1303) presented in various embodimentsof the present invention is novel, logistically simple, and costeffective, as the target subject is required to rotate 360-degrees infront of a unique camera equipment (i.e. RGB-D camera) capable ofcapturing not only color data but also real-time depth parameters (i.e.z-axis for three-dimensional depth perception) of the target subject. Inone embodiment of the invention, one or more distance-measuring lasersmay be incorporated into the RGB-D camera to measure accurate depths ofthe target subject. In another embodiment of the invention, infrared orultrasonic sensors may be incorporated into the RGB-D camera todetermine approximate depths of the target subject, even if thedepth-determination data is not as finely granular as laser-basedmeasurements. Yet in another embodiment of the invention, a multiplenumber of lenses may be incorporated into the RGB-D camera to providedepth measurements of the target subject.

The self-rotation provided by the target subject and the intelligentmachine-learning of volumetric extrapolations, estimations, andrefinements in a hologram production feedback loop, as previously shownin FIGS. 11-12, enable this embodiment of the present invention tosynthesize a hologram that represents the target subject far moreconveniently and cost-effectively, compared to the conventionalhologram-generating method (1301). In the cost and convenience advantageexample (1300) illustrated in FIG. 13, the single RGB-D camera andmachine learning-based holographic image capture method with the targetsubject's 360-degree self-rotation in front of the RGB-D camera (1303)may be up to 90 percent cheaper than the conventionalhologram-generating method (1301) that traditionally utilizes finelypre-calibrated multi-angle cameras prior to image capture.

FIG. 14 shows a novel real-world object holographic transport andcommunication room system configuration (1400), in accordance with anembodiment of the invention. The system configuration in this embodimentcomprises a real-world object holographic transport and communicationroom (1411) with one or more vertical walls, a hologram bilateralmonitoring device (1403) mounted on a vertical wall of the room, asingle RGB-D camera (1405) installed near the hologram bilateralmonitoring device (1403), a mixed-reality (MR) headset (1401) worn by ahologram user, a hologram (1407) visible through the MR headset (1401),and an autostereoscopic holographic display and capture tubular device(1409) that does not require a separate headset gear to visualize thehologram, as shown in the novel real-world object holographic transportand communication room system configuration (1400).

Preferably, the real-world object holographic transport andcommunication room (1411) also incorporates a holographic visualizationtable to place the autostereoscopic holographic display and capturetubular device (1409) on top of the table's surface, as shown in FIG.14. The equipment (e.g. 1401, 1403, 1405, 1407, 1409) installed in eachholographic transport and communication room is standardized across thesame types of rooms constructed in various locations. Similarly, thedimensions (e.g. 2.5 m×2.5 m) of each real-world object holographictransport and communication room (1411) are standardized as a “singlecell” model unit that can be replicated in physical constructions ofholographic transport and communication rooms as “multiple cells” thatare operatively connected across even great distances (e.g. 1411, 1413,1415) to formulate a holographic transport and communication ecosystemcomprising a numerous and scalable number of the real-world objectholographic transport and communication rooms, using a 5G wireless datanetwork and/or another broadband network, as shown in the novelreal-world object holographic transport and communication room systemconfiguration (1400).

In a preferred embodiment of the invention, the real-world objectholographic transport and communication room (1411) may be an enclosedroom or a booth with one or more vertical walls, which are predefined instandardized dimensions with installation plans that are also predefinedwith a standardized suite of electronic equipment for installationwithin the enclosed room or the booth structure. In the particularreal-world object holographic transport and communication room systemconfiguration (1400) as shown in FIG. 14, this standardized room orbooth space is approximately 2.5 meters wide and 2.5 meters long, andcan accommodate one to three users to conduct or experience holographictransports and communications with remotely-located people in otherstandardized real-world object holographic transport and communicationroom systems. By performing a 360-degree self-rotation in front of thesingle RGB-D camera (1405), each user is also able to create one's ownhologram representing his or her appearance in the real-world objectholographic transport and communication room (1411), which eithercontains or operatively connected to a graphics processing serverexecuting a real-time full 4D dynamic reconstruction module thatcreates, estimates, extrapolates, and refines (i.e. sharpens, improves,etc.) the user's holographic representations based on a machine-learningfeedback loop during the user's 360-degree self-rotation sequence. Inaddition, the 360-degree self-rotation method in front of the singleRGB-D camera (1405) may also be utilized to create and record a user'sown 4D holographic contents inside the room, even if no real-timeholographic communication is invoked with another person in anotherholographic transport and communication room.

Furthermore, the hologram bilateral monitoring device (1403) isconfigured to provide a simultaneous and juxtaposed (i.e. side-by-side)visualization of a holographic representation of the user standing infront of the single RGB-D camera (1405), while also displaying, inreal-time, another holographic representation of the user'scommunication partner, who is remotely located outside the real-worldobject holographic transport and communication room (1411). In addition,the MR headset (1401) worn by another user in the room enables in-roomvisualization of another remotely-located communication partner as thehologram (1407) visible through the MR headset (1401). Preferably, thehologram (1407) visible through the MR headset (1401) is a life-size1-to-1 ratio representation of the remotely-located communicationpartner.

As shown in FIG. 14, if the user is not wearing the MR headset (1401),he or she is also able to visualize that remotely-located communicationpartner as an autostereoscopic hologram generated inside theautostereoscopic holographic display and capture tubular device (1409).Users in one real-world object holographic transport and communicationroom are also empowered to collaborate in mixed-reality collaborativedecoration of a hologram in real-time with other users in otherreal-world object holographic transport and communication rooms, whichare located remotely around the world.

Moreover, users inside the real-world object holographic transport andcommunication room (1411) are also able to search and retrieve recorded(i.e. non real-time) holograms and holographic contents to experiencerecorded 4D holographic concerts, shows, and sporting events, inaddition to being able to participate in real-time holographiccommunications with other users in other real-world object holographictransport and communication rooms installed around the world. Therecorded holographic contents may be free, pay-per-view, or subscriptionview-based.

One key advantage of this novel real-world object holographic transportand communication room system configuration (1400) is the compactness ofthe required space. Because only one specialty RGB-D camera is utilizedin the target object image capture, instead of conventional multi-angledcameras surrounding the target object, the real-world object holographictransport and communication room can be designed in a tight space, whilecompletely eliminating the need for cumbersome multiple camera anglecalibrations that require inefficient time and effort in conventionalhologram generation methods. Another key advantage of this novelreal-world object holographic transport and communication room systemconfiguration (1400) is the cost efficiency related to the reducednumber of cameras and the graphics processing servers required inreal-time hologram synthesis, which is made possible by instructing eachuser to self-rotate himself or herself 360-degrees in front of onespecialty RGB-D camera, and by executing a novel real-time full 4Ddynamic reconstruction module that creates, estimates, extrapolates, andrefines the user's holographic representations based on amachine-learning feedback loop during the user's 360-degreeself-rotation sequence, as previously illustrated in FIGS. 11 and 12.

FIG. 15 shows another novel real-world object holographic transport andcommunication room system configuration (1500), in accordance with anembodiment of the invention. The system configuration in this embodimentcomprises a real-world object holographic transport and communicationroom (1513) with one or more vertical walls, a hologram bilateralmonitoring device (1505) mounted on a vertical wall of the room, asingle RGB-D camera (1503) installed near the hologram bilateralmonitoring device (1505), a mixed-reality (MR) headset worn by ahologram user, a hologram (1509) visible through the MR headset, an MRcontent synthesis table (1507) designed to superimpose a holographicmixed-reality (HMR) environment on top of the table's surface, and alife-size (i.e. 1-to-1 ratio) autostereoscopic holographic display andcapture tubular device (1511) that does not require a separate headsetgear to visualize a life-size hologram, as shown in this novelreal-world object holographic transport and communication room systemconfiguration (1500).

Preferably, the MR content synthesis table (1507) is utilized by aplurality of users in various holographic transport and communicationrooms around the world to collaborate on a creation of a bilateral,multilateral, and/or decorative holographic content, with changes madeto the content being reflected on the holographic visualizations in realtime on top of the table's surface. The equipment (e.g. 1503, 1505,1507, 1511) installed in each holographic transport and communicationroom is standardized across the same types of rooms constructed invarious locations. Similarly, the dimensions (e.g. 3.5 m×3.5 m) of eachreal-world object holographic transport and communication room (1513)are standardized as a “single cell” model unit that can be replicated inphysical constructions of holographic transport and communication roomsas “multiple cells” that are operatively connected across even greatdistances (e.g. 1513, 1515, 1517) to formulate a holographic transportand communication ecosystem comprising a numerous and scalable number ofthe real-world object holographic transport and communication rooms,using a 5G wireless data network and/or another broadband network, asshown in the novel real-world object holographic transport andcommunication room system configuration (1500).

In a preferred embodiment of the invention, the real-world objectholographic transport and communication room (1513) may be an enclosedroom or a booth with one or more vertical walls, which are predefined instandardized dimensions with installation plans that are also predefinedwith a standardized suite of electronic equipment for installationwithin the enclosed room or the booth structure. In the particularreal-world object holographic transport and communication room systemconfiguration (1500) as shown in FIG. 15, this standardized room orbooth space is approximately 3.5 meters wide and 3.5 meters long, andcan accommodate up to five users simultaneously to conduct or experienceholographic transports and communications with remotely-located peoplein other standardized real-world object holographic transport andcommunication room systems. By performing a 360-degree self-rotation infront of the single RGB-D camera (1503), each user is also able tocreate one's own hologram representing his or her appearance in thereal-world object holographic transport and communication room (1513),which either contains or operatively connected to a graphics processingserver executing a real-time full 4D dynamic reconstruction module thatcreates, estimates, extrapolates, and refines (i.e. sharpens, improves,etc.) the user's holographic representations based on a machine-learningfeedback loop during the user's 360-degree self-rotation sequence. Inaddition, the 360-degree self-rotation method in front of the singleRGB-D camera (1503) may also be utilized to create and record a user'sown 4D holographic contents inside the room, even if no real-timeholographic communication is invoked with another person in anotherholographic transport and communication room.

Furthermore, the hologram bilateral monitoring device (1505) isconfigured to provide a simultaneous and juxtaposed (i.e. side-by-side)visualization of a holographic representation of a user (1501) standingin front of the single RGB-D camera (1503), while also displaying, inreal-time, another holographic representation of the user'scommunication partner, who is remotely located outside the real-worldobject holographic transport and communication room (1513). In addition,the MR headset worn by another user in the room enables in-roomvisualization of another remotely-located communication partner as thehologram (1509) visible through the MR headset. Preferably, the hologram(1509) visible through the MR headset is a life-size 1-to-1 ratiorepresentation of the remotely-located communication partner.

As shown in FIG. 15, if a user is not wearing the MR headset, he or sheis also able to visualize that remotely-located communication partner asan autostereoscopic hologram generated inside the life-sizeautostereoscopic holographic display and capture tubular device (1511).Furthermore, the life-size autostereoscopic holographic display andcapture tubular device (1511) preferably also integrates a single RGB-Dcamera that can capture and generate a hologram of a person or an objectstanding directly in front of the life-size autostereoscopic holographicdisplay and capture tubular device (1511), as shown in this real-worldobject holographic transport and communication room system configuration(1500). Then, the newly-generated hologram based on a 360-degreeself-rotation of the target object in front of the life-sizeautostereoscopic holographic display and capture tubular device (1511)is transmitted to and displayed in another real-world object holographictransport and communication room as part of the real-time holographiccommunication. Users in one real-world object holographic transport andcommunication room are also empowered to collaborate in mixed-realitycollaborative decoration of a hologram in real-time with other users inother real-world object holographic transport and communication rooms,which are located remotely around the world.

Moreover, users inside the real-world object holographic transport andcommunication room (1513) are also able to search and retrieve recorded(i.e. non real-time) holograms and holographic contents to experiencerecorded 4D holographic concerts, shows, and sporting events, inaddition to being able to participate in real-time holographiccommunications with other users in other real-world object holographictransport and communication rooms installed around the world. Therecorded holographic contents may be free, pay-per-view, or subscriptionview-based.

One key advantage of this novel real-world object holographic transportand communication room system configuration (1500) is the compactness ofthe required space. Because only one specialty RGB-D camera is utilizedin the target object image capture, instead of conventional multi-angledcameras surrounding the target object, the real-world object holographictransport and communication room can be designed in a tight space, whilecompletely eliminating the need for cumbersome multiple camera anglecalibrations that require inefficient time and effort in conventionalhologram generation methods. Another key advantage of this novelreal-world object holographic transport and communication room systemconfiguration (1500) is the cost efficiency related to the reducednumber of cameras and the graphics processing servers required inreal-time hologram synthesis, which is made possible by instructing eachuser to self-rotate himself or herself 360-degrees in front of onespecialty RGB-D camera, and by executing a novel real-time full 4Ddynamic reconstruction module that creates, estimates, extrapolates, andrefines the user's holographic representations based on amachine-learning feedback loop during the user's 360-degreeself-rotation sequence, as previously illustrated in FIGS. 11 and 12.

FIG. 16 shows a two-room application example (1600) of the novelreal-world object holographic transport and communication room system,in accordance with an embodiment of the invention. In this example(1600), a first real-world object holographic transport andcommunication room (1601) is located in Seoul, and a second real-worldobject holographic transport and communication room (1603) is located inBerlin. These two rooms are operatively connected by one or morebroadband wireless and/or wired data networks, such as 5G, opticalcable, and/or other high-speed networks.

As shown in FIG. 16, users in the two remotely-located holographictransport and communication rooms are able to visualize, collaborate,and share holograms during their joint holographic mixed-reality (HMR)projects and/or real-time holographic communications. By utilizing thenovel 360-degree self-rotation sequence in front of a single RGB-Dcamera, this embodiment of the present invention can obsolete themulti-camera pre-calibration and exorbitant space requirements ofconventional hologram-generation equipment per room, and instead provideeasier holographic communication system implementations, reduced spacerequirements, and initial startup and operational cost savings inmultiple geographic locations.

FIG. 17 shows a system component diagram (1700) for the two-roomapplication example of the novel real-world object holographic transportand communication room system, in accordance with an embodiment of theinvention. As shown in this illustration, fast wireless and/or wiredlocal area network with CAT 6 cabling for gigabit throughput, virtualreality-ready 60 GHz wireless routers, and/or 5G wide-area networkinfrastructure are preferable for implementing the novel real-worldobject holographic transport and communication room system for seamlesshigh-resolution real-time bilateral or multilateral holographiccommunication. By utilizing the novel 360-degree self-rotation sequencein front of a single RGB-D camera, this embodiment of the presentinvention can remove the need for multi-camera pre-calibration andexorbitant space requirements of conventional hologram-generationequipment for each room. Instead, various embodiments of the presentinvention provide easier holographic communication systemimplementations, reduced space requirements per holographic transportand communication room, and initial startup and operational cost savingsin multiple geographic locations.

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of thisdisclosure, will appreciate that other embodiments can be devised whichdo not depart from the scope of the invention as disclosed herein.Accordingly, the scope of the invention should be limited only by theattached claims provided herein.

What is claimed is:
 1. A real-world object holographic transport and communication room system comprising: a holographic transport and communication room with a vertical wall; a hologram bilateral monitoring device mounted on the vertical wall; a single red-green-blue (RGB)-depth (RGB-D) camera installed with the hologram bilateral monitoring device, wherein the single RGB-D camera captures real-time z-axis depth parameters of a target object, in addition to conventional RGB color data; the target object standing and self-rotating 360-degrees at least once in front of the single RGB-D camera to enable the single RGB-D camera to capture three-dimensional (3D) volume information of the target object over a specified duration; a graphics server receiving a continuous stream of the 3D volume information of the target object over the specified duration while the target object is self-rotating 360-degrees at least once in front of the single RGB-D camera, wherein the specified duration of the continuous stream of the 3D volume information provides time-variable volumetric information of the target object required to create, sharpen, and display a computerized hologram of the target object by the graphics server in a real-time bilateral holographic communication with a remote user outside the holographic transport and communication room, wherein the 3D volume information of the target object undergoes computerized graphical transformations involving relighting, subject depth calculations, geometrical extrapolations, and volumetric reconstructions through machine learning by the graphics server to create, dynamically sharpen, and display the computerized hologram of the target object over the specified duration, while the target object is self-rotating 360-degrees at least once in front of the single RGB-D camera; a mixed-reality (MR) headset worn by a local user located inside the holographic transport and communication room; a remote hologram from the remote user projected in the holographic transport and communication room, wherein the remote hologram from the remote user is visible through the MR headset worn by the local user inside the holographic transport and communication room; and an autostereoscopic holographic display and capture tubular device that does not require a separate headset gear to visualize the remote hologram for other local users in the holographic transport and communication room.
 2. The real-world object holographic transport and communication room system of claim 1, further comprising a mixed-reality (MR) content synthesis table configured to superimpose a holographic mixed-reality (HMR) environment on top of the MR content synthesis table's surface, wherein the HMR environment invites multiple local and remote user participations from various holographic transport and communication rooms around the world to collaborate on a creation of a bilateral or multilateral holographic content in real time, with changes made to the bilateral or multilateral holographic content being reflected on top of the MR content synthesis table's surface in real time.
 3. The real-world object holographic transport and communication room system of claim 1, further comprising a holographic visualization table to place the autostereoscopic holographic display and capture tubular device on top of the holographic visualization table's surface.
 4. The real-world object holographic transport and communication room system of claim 1, wherein the autostereoscopic holographic display and capture tubular device is a 1-to-1 ratio autostereoscopic tube that provides the remote hologram in undistorted original dimensions without miniaturization or magnification.
 5. The real-world object holographic transport and communication room system of claim 1, wherein the computerized hologram of the target object, which is created and dynamically sharpened while the target object is self-rotating 360-degrees at least once in front of the single RGB-D camera, incorporates a time variable as a fourth dimension to generate a 4-dimensional (4D) hologram, wherein first three dimensions are related to the 3D volume information of the target object at a particular time slice, and wherein the fourth dimension encodes changes or movements of the 3D volume information over the specified duration.
 6. The real-world object holographic transport and communication room system of claim 1, wherein the hologram bilateral monitoring device mounted on the vertical wall displays both the computerized hologram of the target object located in the holographic transport and communication room and the remote hologram from the remote user.
 7. The real-world object holographic transport and communication room system of claim 1, wherein the remote user is located inside a second holographic transport and communication room with a second hologram bilateral monitoring device, a second single RGB-D camera, and a second target object, which is transformed into the remote hologram by the graphics server when transported to the local user located inside the holographic transport and communication room.
 8. The real-world object holographic transport and communication room system of claim 7, wherein the holographic transport and communication room and the second holographic transport and communication room are operatively connected by a 5G network, CAT 6 Ethernet cabling, 600 GHz wireless routers, or a combination thereof that supports high-throughput graphics data transmissions over remote distances.
 9. The real-world object holographic transport and communication room system of claim 7, wherein each holographic transport and communication room is standardized in dimensions and equipment as a single-cell model unit for scalable and convenient replications of holographic transport and communication room networks around the world. 